Information processing system, method of controlling information processing system and storage medium

ABSTRACT

An information processing system includes: a plurality of processors configured to cause a virtual machine on each processor to be operated or terminated; a plurality of storage devices; and a control device configured to: collect processing loads of the respective processors, cause a first virtual machine which operates on a first processor in which a frequency of a processing load exceeding a first threshold exceeds a first proportion, to be terminated, cause a second processor which is stopped among the plurality of processors, to be operated, and cause a first storage device which is coupled with the terminated first virtual machine among the plurality of storage devices, to be coupled with the operated second processor.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2014-090271, filed on Apr. 24,2014, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to an informationprocessing system, a method of controlling the information processingsystem, and a storage medium.

BACKGROUND

When a virtual computation machine defined on a computation machine ismoved to another computation machine, information used in a processexecuted by the virtual computation machine is sent to an externalstorage device and then is moved to the other computation machine.

In a system including a plurality of server devices, each executing aprocess based on a request, session information included in a serverdevice having a load larger than that of another server device is movedto another server device so as to decentralize the load.

When a process of a virtual computation machine operated on acomputation machine is executed by a virtual computation machine onanother computation machine, configuration information on a storageinterface and the like of the original virtual computation machine isset in the virtual computation machine defined on the other computationmachine, and then the virtual computation machine defined on the othercomputation machine is operated. For example, the original virtualcomputation machine is stopped and a new virtual computation machine issuppressed from starting an operation during a time period whenconfiguration information is exchanged between the original virtualcomputation machine and the new virtual computation machine.

Examples of the related art include Japanese Laid-open PatentPublication No. 2009-145931, Japanese Laid-open Patent Publication No.2007-219964, and Japanese Laid-open Patent Publication No. 2010-33404.

SUMMARY

According to an aspect of the invention, an information processingsystem includes: a plurality of processors configured to cause a virtualmachine on each processor to be operated or terminated; a plurality ofstorage devices; and a control device configured to: collect processingloads of the respective processors, cause a first virtual machine whichoperates on a first processor in which a frequency of a processing loadexceeding a first threshold exceeds a first proportion, to beterminated, cause a second processor which is stopped among theplurality of processors, to be operated, and cause a first storagedevice which is coupled with the terminated first virtual machine amongthe plurality of storage devices, to be coupled with the operated secondprocessor.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an embodiment of an information processing system, acontrol method of the information processing system, and a controlprogram of the information processing system;

FIG. 2 illustrates an example of a state of the information processingapparatus different from a state illustrated in FIG. 1;

FIG. 3 illustrates an example in which a control device collects loadinformation;

FIG. 4 illustrates an example of frequency of load information in eachphysical server, collected by the control device;

FIG. 5 illustrates an example of an operation of the informationprocessing system;

FIG. 6 illustrates an example of change of a load in an operation of theinformation processing system;

FIG. 7 illustrates another example of change of a load in an operationof the information processing system;

FIG. 8 illustrates an example of an operating flow in the controldevice;

FIG. 9 illustrates an example of an ST0 control routine;

FIG. 10 illustrates an example of an ST1 control routine;

FIG. 11 illustrates an example of an ST2 control routine;

FIG. 12 illustrates an example of an ST3 control routine;

FIG. 13 illustrates an example of an ST4 control routine;

FIG. 14 illustrates an example of an ST5 control routine;

FIG. 15 illustrates an example of a program of executing the ST0 controlroutine;

FIG. 16 illustrates an example of a program of executing the ST1 controlroutine;

FIG. 17 illustrates an example of a program of executing the ST2 controlroutine;

FIG. 18 illustrates an example of a program of executing the ST3 controlroutine;

FIG. 19 illustrates an example of a program of executing the ST4 controlroutine;

FIG. 20 illustrates an example of a program of executing the ST5 controlroutine;

FIG. 21 illustrates an example in which a control device changes thestate of an information processing apparatus in another embodiment ofthe information processing system, the control method of the informationprocessing system, and the control program of the information processingsystem; and

FIG. 22 illustrates an example of a relationship between each process, aload, and frequency in another embodiment.

DESCRIPTION OF EMBODIMENTS

For example, when a server device executing information processing isswitched to another server device or when a server device is added, aprocess of switching or adding causes a load of the server device toincrease. For this reason, it is preferable that switching and adding ofa server device are executed before a load of a server device exceeds apredetermined threshold, in order to suppress degradation of a serviceprovided by the server device. It is preferable that a type of aswitched server device or an added server device is selected inaccordance with the extent that a load of the server device changes.

The object of an information processing system, a control method of theinformation processing system, and a recording medium in which a controlprogram of the information processing system is recorded, according tothis disclosure is to change processing performance of the informationprocessing system depending on a change of a load without an influenceon a service provided by the information processing system.

Hereinafter, embodiments will be described using the drawings.

FIG. 1 illustrates an embodiment of an information processing system, acontrol method of the information processing system, and a controlprogram of the information processing system. An information processingsystem SYS1 according to this embodiment includes an informationprocessing apparatus IPE1 and a control device CNTL configured tocontrol the information processing apparatus IPE1. The informationprocessing apparatus IPE1 includes a plurality of servers P (P0, P1, P2,P3, P4, and P5), a switch device SW, and a disk pool DPL. For example,each server P and the control device CNTL are coupled with each otherthrough a network switch NSW such as Ethernet (registered trademark).For example, each server P and the control device CNTL are coupled witha user terminal TM which uses the information processing system SYS1 asa network storage, through the network switch NSW. For example, the userterminal TM is coupled with the information processing system SYS1through a network NW such as the Internet or an intranet. For example,the information processing system SYS1 functions as a storage server ora data center for holding information such as data, transmitted from theuser terminal TM. For example, information transmitted from the userterminal TM may be stored in duplicate in each of a plurality of storagedevices D (for example, three storage devices D).

Each server P is operated based on control of the control device CNTLand is stopped based on control of the control device CNTL. The serversP0, P1, and P2 indicated by solid lines represent the operating serversand the servers P3, P4, and P5 indicated by dot lines represent thestopped servers. The currently stopped servers P3, P4, and P5 are to beoperated based on an instruction from the control device CNTL. Forexample, the currently stopped servers P3, P4, and P5 are in a statewhere power is cut off. The number of servers P is not limited to six.The server P is an example of an information processing section.

Each server P includes a processor such as a central processing unit(CPU) and a storage device such as a memory MEM for storing an operatingsystem (OS) executed by the processor and a program which is forimplementing a function of the server. Each server P may include astorage device such as a hard disk drive (HDD), for storing the program.The OS or the program executed by each server P may be stored in thestorage device D disposed in the disk pool DPL and may be transmitted tothe memory MEM of the server P based on an operating instructionreceived from the control device CNTL.

Each server P includes a monitoring section LDM for monitoring its ownload. The monitoring section LDM is, for example, a load monitoringagent. The monitoring section LDM monitors a load of the CPU at apredetermined interval and notifies the control device CNTL of themonitored load as load information indicating a load of each server P.For example, a “load of a CPU” refers to CPU usage (for example, anaverage value) monitored by the monitoring section LDM for apredetermined period (for example, one second). The monitoring sectionLDM may monitor input/output (I/O) usage (for example, an access rate ofthe storage device D) instead of the CPU usage. The monitoring sectionLDM may be implemented through a program which is executed by the CPUand may be implemented by hardware. Additionally, the monitoring sectionLDM may be implemented by combining a program and hardware.

A plurality of virtual machines V (V0 and V1) for implementing afunction of the server using software may be mounted in each server P.FIG. 1 illustrates an example in which the virtual machines V0 and V1are operated on the server P0. The number of virtual machines V allowedto be operated on each server P is not limited to two. In the followingdescription, the server P physically mounted in the informationprocessing system SYS1 is referred to as a physical server P and thevirtual machine V installed on the physical server P is referred to as avirtual server V.

The disk pool DPL includes a plurality of storage devices D (D0 to D7).For example, the storage device D refers to an HDD or a solid statedrive (SSD), but may refer to other storage devices capable of holdinginformation in a non-volatile manner. For example, the storage devicesD0 to D4 indicated by solid lines represent a state where power issupplied and information such as data is written and read. The storagedevices D5 to D7 indicated by dot lines represent a state where thepower is cut off and information is not written and read. For example,information such as data is deleted in the storage devices D5 to D7indicated by dot lines. The power may be supplied to the storage devicesD5 to D7 in which information is not written and read. For example,supply and cutting of the power in the storage devices D0 to D7 iscontrolled by the control device CNTL. The number of the storage devicesD0 to D7 is not limited to eight. The storage device D is an example ofa storage section.

The switch device SW couples the storage device D with the physicalserver P or the virtual server V based on control of the control deviceCNTL. A dot line seen inside the switch device SW indicates a state inwhich the storage device D is coupled with the physical server P or thevirtual server V. For example, the switch device SW may couple anarbitrary storage device D with the physical server P or the virtualserver V based on control from the control device CNTL. In FIG. 1, foreasily understandable description, one storage device D is coupled withone physical server P or one virtual server V. However, a plurality ofstorage devices D may be coupled with one physical server P or onevirtual server V. The switch device SW is an example of a couplingsection. A disk area network or a storage area network (SAN) forcoupling the storage device D with the physical server P or the virtualserver V is constructed in the information processing apparatus IPE1 byusing the switch device SW.

The switch device SW has performance sufficient for coupling the storagedevice D with the physical server P or the virtual server V without aband (data transmission speed) of the storage device D being reduced.The storage device D is operated as a direct attached storage (DAS).That is, performance of the storage device D which is coupled with thephysical server P or the virtual server V through the switch device SWis equal to performance of a storage device such as an HDD, mounted inthe physical server P. Since the virtual server V is implemented througha program which is executed by the physical server P, access efficiencyof the storage device D by the virtual server V is lower than accessefficiency of the storage device D by the physical server P.

The control device CNTL includes a processor such as a CPU and a storagedevice such as a memory MEM, for storing an OS executed by the processorand a program which is for implementing a function of the control deviceCNTL. The program which is stored in the memory MEM and executed by theCPU of the control device CNTL refers to a control program forcontrolling an operation of the information processing system SYS1. Thecontrol method of the information processing system SYS1 is implementedthrough the program which is stored in the memory MEM and executed bythe CPU of the control device CNTL.

The control device CNTL includes a holding section HLD for holding theload information of each server P transferred from the monitoringsection LDM of the server P. For example, the holding section HLD may beimplemented by using registers mounted in the CPU of the control deviceCNTL and may be held in the memory MEM of the control device CNTL. Thecontrol device CNTL may include a storage device such as an HDD, forstoring a program.

In the example illustrated in FIG. 1, the physical servers P0, P1, P2are operated. The storage device D0 is coupled with the physical serverP0, the storage device D3 is coupled with the physical server P1, andthe storage device D4 is coupled with the physical server P2. Thevirtual servers V0 and V1 are operated on the physical server P0, thestorage device D1 is coupled with the virtual server V0, and the storagedevice D2 is coupled with the virtual server V1. In FIG. 1, the physicalservers P0, P1, and P2 and the virtual servers V0 and V1 operate fiveservers and the information processing system SYS1 is operated as astorage server or a data center, for example.

The monitoring section LDM of each of the physical servers P0, P1, andP2 in operation monitors a load of the CPU periodically and transmitsthe load information indicating the monitored load to the control deviceCNTL. The control device CNTL holds the received load information in theholding section HLD. The control device CNTL controls to switch thevirtual server V to the physical server P or to switch the physicalserver P to the virtual server V, based on the load information of eachof the physical servers P0, P1, and P2 in operation, which is held inthe holding section HLD. The control device CNTL controls to add avirtual server V, to add a physical server P, or to delete a virtualserver V. The control method of the information processing system SYS1executed by the control device CNTL will be described using FIGS. 3 to20.

For example, the control device CNTL controls to equally writeinformation such as file data and image data, which is transmitted fromthe user terminal TM in the storage devices D respectively coupled withservers which include the physical server P and the virtual server V.For example, the storage devices D respectively coupled with serverswhich include the physical server P and the virtual server V aredisposed in a hash space evenly. In other words, the control device CNTLassigns information to the servers such as data such that loads of thephysical server P in operation and the virtual server V in operation areequal to each other. A parent server which is any one of the physicalservers P in operation may assign information such as data (that is,designation of a server which transmits information such as data).Information including data and the like is simply referred to as databelow.

Access performance of the virtual server V accessing the storage deviceD is lower than access performance of the physical server P accessingthe storage device D. For this reason, a load of the virtual server Vmay be smaller than a load of the physical server P by the number ofstorage devices D coupled with the virtual server V being less than thenumber of the storage device D coupled with the physical server P. Aload of the virtual server V may be smaller than a load of the physicalserver P by a hash space allocated to the storage device D coupled withthe virtual server V being smaller than a hash space allocated to thestorage device D coupled with the physical server P.

FIG. 2 illustrates an example of a state of the information processingapparatus IPE1 different from a state of the information processingapparatus IPE1 illustrated in FIG. 1. For example, in FIG. 2, thevirtual server V0 (FIG. 1) which has operated on the physical server P0is removed, the stopped physical server P3 starts to be operated, andthe storage device D1 which has been coupled with the virtual server V0is coupled with the physical server P3. That is, FIG. 2 illustrates anexample in which the virtual server V0 (FIG. 1) is switched to thephysical server P3 based on a change of a load in any one of thephysical servers P0, P1, and P2. A rectangle indicated by a dot line inthe physical server P0 means that a program for implementing the virtualserver V0 illustrated in FIG. 1 is terminated and an operation of thevirtual server V0 on the physical server P0 is terminated.

Switching the virtual server V0 to the physical server P3 is performedby exchanging configuration information (for example, a host name whichis a unique name of the server and an internet protocol (IP) address) ofthe physical server P3 and configuration information of the virtualserver V0, for example. At this time, the control device CNTL controlsthe switch device SW to separate the storage device D1 from the virtualserver V0 and to couple the separated storage device D1 with thephysical server P3.

For example, the control device CNTL changes a state of the informationprocessing apparatus IPE1 to a state illustrated in FIG. 2 from a stateillustrated in FIG. 1 when a frequency of a load in any one of thephysical servers P0 to P2 to the maximum load applied to each physicalserver P being more than 30% and equal to or less than 80% exceeds apredetermined proportion. For example, the frequency refers to aproportion of the number of pieces of load information which indicates aload of more than 30% and equal to or less than 80% to the number ofpieces of load information which is held in the holding section HLD andis transmitted to each of the physical servers P0, P1, and P2.

FIG. 3 illustrates an example in which the control device CNTLillustrated in FIG. 1 collects load information. For example, each barof a bar graph illustrated on an upper side of FIG. 3 indicates loadinformation of which the control device CNTL is notified by onemonitoring section LDM of the physical server P in operation. In theexample illustrated in FIG. 3, the control device CNTL holds 100 piecesof load information received by the physical server P in the holdingsection HLD. The load information indicates a load (any value from 0% to100%) of the CPU mounted in the physical server P performingnotification of the load information. In FIG. 3, for easilyunderstandable description, the load information is indicated by the bargraph. However, the server P notifies the control device CNTL of theload information as a load value in practice. For example, themonitoring section LDM notifies the control device CNTL of the loadinformation once a second. For this reason, a time count TC which isindicated on a horizontal axis of FIG. 3 and refers to the number ofnotifications of the load information also indicates the number ofseconds.

The control device CNTL compares load values indicated by the loadinformation held in the holding section HLD with threshold values VT1,VT2, and VT3 each time load information is received 100 times from eachoperating physical server P. For example, the threshold value VT1 refersto 80%, the threshold value VT2 refers to 30%, and the threshold valueVT3 refers to 2%. The threshold values VT1, VT2, and VT3 may refer toother values as long as the threshold values VT1, VT2, and VT3 satisfy arelationship of VT1>VT2>VT3 and the number of pieces of load informationto be compared is not limited to 100. For example, in an example of aprocessing flow illustrated in FIGS. 9 to 14, the control device CNTLcompares each load value indicated by each piece of load informationheld in the holding section HLD with the threshold values VT1, VT2, andVT3 each time the load information is received 600 times (that is, foreach 600 seconds).

The control device CNTL increases a variable LUTL by “one” when a loadindicated by the load information is less than the threshold value VT3.The control device CNTL increases a variable MUTL by “one” when a loadindicated by the load information is more than the threshold value VT2and equal to or less than the threshold value VT1. The control deviceCNTL increases a variable HUTL by “one” when a load indicated by theload information exceeds the threshold value VT1. The variables LUTL,MUTL, and HUTL are assigned to each physical server P.

The values of variables LUTL, MUTL, and HUTL are initialized to “0”before the load information is collected. For example, when the controldevice CNTL collects the load information in a unit of 100 seconds(TC=100), the variables LUTL, MUTL, and HUTL are initialized for each100 seconds. In the following description, a region of exceeding thethreshold value VT1 (80%) is also referred to as a high load region, aregion of more than the threshold value VT2 (30%) and equal to or lessthan the threshold value VT1 (80%) is also referred to as a middle loadregion, and a region of less than the threshold value VT3 (2%) is alsoreferred to as a low load region.

In FIG. 3, the variable LUTL has a value of “5”, the variable MUTL has avalue of “82”, and the variable HUTL has a value of “9” in a state wherethe control device CNTL collects the load information 100 times. Thecontrol device CNTL divides the values of the variables LUTL, MUTL, andHUTL by a value (=100) of the time count TC to obtain the frequency ofthe load (ratio of time) at each of the high load region, the middleload region, and the low load region for a predetermined period (forexample, 100 seconds). For example, in the example illustrated in FIG.3, the frequency of the load at the high load region is “0.09”, thefrequency of the load at the middle load region is “0.82”, and thefrequency of the load at the low load region is “0.05”. The frequency ofthe load refers to a proportion of loads distributed at a load regionhaving a predetermined range. A relationship of the frequency of theload indicated by the variables LUTL, MUTL, and HUTL which are assignedto each of the plurality of physical servers P which are operating willbe described with reference to FIG. 4.

As illustrated in FIGS. 9 to 14, the control device CNTL determineswhether or not a result of collecting the load information satisfies thefollowing five conditions of Condition A to Condition E.

-   -   (Condition A) the frequency of the load at the middle load        region exceeds “0.8”.    -   (Condition B) the frequency of the load at the middle load        region exceeds “0.7”    -   (Condition C) the frequency of the load at the high load region        exceeds “0.9”.    -   (Condition D) the frequency of the load at the low load region        exceeds “0.8”.    -   (Condition E) the frequency of the load at the low load region        exceeds “0.9”.

The control device CNTL controls the physical server P, the virtualserver V, and the switch device SW based on a state of the informationprocessing apparatus IPE1 including information such as the number ofthe operating physical servers P and the number of the operating virtualservers V. FIG. 5 illustrates an example in which the control deviceCNTL controls the physical server P, the virtual server V, and theswitch device SW. As illustrated in FIG. 5, the condition used fordetermination is different in accordance with states ST0 to ST5 of theinformation processing apparatus IPE1.

FIG. 4 illustrates an example of the frequency of the load informationof each of the physical servers P0, P1, and P2, collected by the controldevice CNTL illustrated in FIG. 1. FIG. 4 illustrates an example of thefrequency of the load information when the control device CNTL detectsthat (Condition A) illustrated in FIG. 3 is satisfied and changes astate of the information processing apparatus IPE1 to the stateillustrated in FIG. 2 from the state illustrated in FIG. 1.

For example, when the storage devices D are disposed evenly using a hashspace, it is highly likely that changes of the load in the respectivephysical servers P in operation have a tendency to be similar to eachother. For this reason, the frequency of the load of one operatingphysical server P at the low load region, the middle load region, andthe high load region has a similar tendency as the frequency of the loadof another operating physical server P at the low load region, themiddle load region, and the high load region. For example, when thefrequency of the load of one operating physical server P at the middleload region is high, the frequency of the load of another operatingphysical server P at the middle load region also is high.

The load of the physical server P including the operating virtual serverV is higher than the load of the physical server P not including thevirtual server V. For example, the frequency of the load at each of thehigh load region, the middle load region, and the low load region in thephysical server P0 causing the two virtual servers V0 and V1 to beoperated in the state illustrated in FIG. 1 is higher than the frequencyof the load at each of the high load region, the middle load region, andthe low load region in the others physical servers P1 and P2. For thisreason, for example, it is much likely that each of (Condition A),(Condition B), and (Condition C) illustrated in FIG. 3 is satisfiedbased on the frequency of the load of the physical server P causing thevirtual server V to be operated.

FIG. 5 illustrates an example of an operation of the informationprocessing system SYS1 illustrated in FIG. 1. That is, FIG. 5illustrates an example of the control method of the informationprocessing system SYS1 performed by the control device CNTL.

In the example illustrated in FIG. 5, the state of the informationprocessing apparatus IPE1 is changed between the states ST0, ST1, ST2,ST3, ST4, and ST5. The state ST0 refers to a state illustrated in FIG. 1and the state ST1 refers to a state illustrated in FIG. 2. When the loadof the physical server P in operation does not satisfy a condition forchanging the state in each of the states ST0 to ST5, the state is held(marks of (a), (b), (c), (d), (e), and (f) in FIG. 5). The operation ofthe information processing system SYS1 is not limited to the exampleillustrated in FIG. 5. That is, the number of the states ST0 to ST5 isnot limited to six and specifications of transitions between the statesare not limited to the example illustrated in FIG. 5.

For example, when the frequency of the load included at the middle loadregion exceeds “0.8” in the state ST0 and in any one of the physicalservers P0 to P2 (Condition A), the state of the information processingapparatus IPE1 is changed to the state ST1 (the mark (g) in FIG. 5). Inthis case, the virtual server V0 which has operated on the physicalserver P0 is terminated, the physical server P3 which has been stoppedstarts to be operated, and the storage device D1 which has been coupledwith the virtual server V0 is coupled with the physical server P3. Thatis, the virtual server V0 is replaced with the physical server P3. Thenumber of servers operated in the information processing apparatus IPE1and in the state ST1 is five and is the same as that in the state ST0.However, performance of the information processing apparatus IPE1 isimproved by replacing the virtual server V0 with the physical server P3.That the performance of the information processing apparatus IPE1 isimproved without changing the number of servers is also referred to asscale-out for performance or is also simply referred to as scale-out inthe following description.

The virtual server V1 may be replaced with the physical server P3 forchanging the state ST0 to the state ST1. When the virtual server V isoperated on the physical server P1 or P2, the virtual server V operatedon the physical server P1 or P2 may be replaced with the physical serverP3.

When the load of the physical server P tends to increase, the controldevice CNTL performs scale-out of replacing the virtual server V0 to thephysical server P3 and thus it is possible to increase a processingcapacity of the information processing apparatus IPE1. Accordingly, itis possible to stably operate the information processing apparatus IPE1even when the load of the information processing apparatus IPE1 furtherincreases after that. Since the storage device D1 which has been coupledwith the virtual server V0 is coupled with the physical server P3,transition of data between the storage devices D does not occur and thusit is possible to hold allocation of the storage device D1 to the hashspace.

The configuration information such as an IP address is changed betweenthe virtual server V0 and the physical server P3 with switching from thevirtual server V0 to the physical server P3. For this reason, anincrement of the load of the information processing apparatus IPE1 byswitching from the virtual server V0 to the physical server P3 issmaller than an increment of the load when data is transitioned. Thescale-out performed by switching the servers is performed in a statewhere the load of the physical server P tends to increase. Thus, anincrease of the load by switching the servers does not have an influenceon a service provided by the information processing system SYS1.Accordingly, an increase of the load of the information processingapparatus IPE1 due to switching the servers is minimized and thus it ispossible to improve processing performance of the information processingsystem SYS1 without an influence on a service provided by theinformation processing system SYS1.

When the frequency of the load included at the middle load regionexceeds “0.8” in the state ST1 and in any one of the physical servers P0to P3 (Condition A), the state of the information processing apparatusIPE1 is changed to the state ST2 (the mark (h) in FIG. 5). In this case,the virtual server V1 which has operated on the physical server P0 isterminated, the physical server P4 which has been stopped starts to beoperated, and the storage device D2 which has been coupled with thevirtual server V1 is coupled with the physical server P4. Switching thevirtual server V1 to the physical server P4 is performed by exchangingconfiguration information (for example, a host name and an IP address)of the physical server P4 and configuration information of the virtualserver V1 and switching the coupling of the storage device D1 from thevirtual server V0 to the physical server P0, for example.

Improvement of performance of the information processing apparatus IPE1caused by performing scale-out from the state ST1 to the state ST2 issimilar to improvement of performance of the information processingapparatus IPE1 caused by changing the state ST0 to the state ST1. It ispossible to obtain the same effect as through scale-out from the stateST0 to the state ST1 for performance by performing scale-out from thestate ST1 to the state ST2 for performance. The virtual server Vreplaced with the physical server P4 is not limited to the virtualserver V1, similarly to the change of the state ST0 to the state ST1.

When the frequency of the load included at the middle load regionexceeds “0.7” in the state ST2 and in any one of the physical servers P0to P4 (Condition B), the state of the information processing apparatusIPE1 is changed to the state ST3 (the mark (i) in FIG. 5). In this case,a new virtual server V2 is operated on the physical server P0 and a newstorage device D5 is coupled with the virtual server V2. That is,scale-out of adding the virtual server V2 to the information processingapparatus IPE1 is performed. Since the new storage device D5 is added,reallocation of the hash space is performed for the storage devices D0to D5 respectively coupled with the physical servers P0 to P4 inoperation and the virtual server V2 in operation, for example. Thereallocation of the hash space results in transition of data between thestorage devices D0 to D5. The virtual server V operated on the physicalservers P1 to P4 may be added to the information processing apparatusIPE1 instead of the virtual server V2, for changing the state ST2 to thestate ST3.

The control device CNTL adds the new virtual server V2 in the load beinglow when a tendency of increasing in the load of the physical server Pis lower than a tendency of increasing in the load when the state ST0 ischanged to the state ST1. Accordingly, it is possible to include the newvirtual server V2 when the load of the information processing apparatusIPE1 is further increased. Since data is transitioned between thestorage devices D in the load being low, it is possible to have areduced influence on the information processing apparatus IPE1 due totransition of data, compared to when the load is high and thus data istransitioned.

When the frequency of the load of any one of the physical servers P0 toP4 included at the high load region exceeds “0.9” in the state ST2(Condition C), the state of the information processing apparatus IPE1 ischanged to the state ST4 (the mark (j) in FIG. 5). In this case, a newphysical server P5 is operated and the new storage device D5 is coupledwith the physical server P5. That is, scale-out of adding the physicalserver P5 to the information processing apparatus IPE1 is performed.Since the new storage device D5 is added, reallocation of the hash spaceis performed for the storage devices D0 to D5 respectively coupled withthe physical servers P0 to P5 in operation, for example. Thereallocation of the hash space results in transition of data between thestorage devices D0 to D5. Improvement of performance of the informationprocessing apparatus IPE1 caused by changing the state ST2 to the stateST4 is higher than improvement of performance of the informationprocessing apparatus IPE1 caused by changing the state ST2 to the stateST3. The other stopped physical server P may be added to the informationprocessing apparatus IPE1 for changing the state ST2 to the state ST4.

The control device CNTL adds a new physical server P having a processingcapacity higher than that of the virtual server V when the load of thephysical server P tends to increase rapidly. With this, it is possibleto transition data between the storage devices D while an influence oftransition of data on the information processing apparatus IPE1 isminimized. The frequency of the load may satisfy Condition A orCondition B ahead of satisfying Condition C, in a normal operation ofthe information processing system SYS1 in many cases. For this reason,for example, the frequency of changing the state ST2 to the state ST4 islower than frequency of changing the state ST2 to the state ST3. A stateof the information processing apparatus IPE1 in the state ST5 is thesame as a state of the information processing apparatus IPE1 in thestate ST4.

When the frequency of the load of any one of the physical servers P0 toP4 included at the middle load region exceeds “0.8” in the state ST3(Condition A), the state of the information processing apparatus IPE1 ischanged to the state ST5 (the mark (k) in FIG. 5). In this case, thevirtual server V2 which has operated on the physical server P0 isremoved, the stopped physical server P5 starts to be operated, and thestorage device D5 which has been coupled with the virtual server V2 iscoupled with the physical server P5, similarly to the change of thestate ST0 to the state ST1. That is, scale-out for performance isperformed.

Switching the virtual server V2 to the physical server P5 is performedby exchanging configuration information (for example, a host name and anIP address) of the physical server P5 and configuration information ofthe virtual server V2 and switching the coupling of the storage deviceD5 from the virtual server V2 to the physical server P5, for example.

When the frequency of the load of any one of the physical servers P0 toP5 included at the low load region exceeds “0.8” in the state ST5(Condition D), the state of the information processing apparatus IPE1 ischanged to the state ST3 (the mark (I) in FIG. 5). In this case, thephysical server P5 is stopped, the virtual server V2 is operated on thephysical server P0, and the storage device D5 which has been coupledwith the physical server P5 is coupled with the virtual server V2. Thatis, the physical server P5 is replaced with the virtual server V2 and astate of the information processing apparatus IPE1 is changed in reverseto being changed from the state ST3 to the state ST5. The number ofservers operated in the information processing apparatus IPE1 in thestate ST4 is six and is the same as that in state ST5. However,performance of the information processing apparatus IPE1 is degraded byreplacing the physical server P5 with the virtual server V2. That theperformance of the information processing apparatus IPE1 is degradedwithout changing the number of servers is also referred to as scale-infor performance or is also simply referred to as scale-in in thefollowing description. The physical server P5 may be replaced with thevirtual server V on the physical servers P1 to P4 in operation forchanging the state ST5 to the state ST3.

Switching the physical server P5 to the virtual server V2 is performedby exchanging configuration information (for example, a host name and anIP address) of the physical server P5 and configuration information ofthe virtual server V2 and switching the coupling of the storage deviceD5 from the physical server P5 to the virtual server V2, for example.

Since the storage device D5 which has been coupled with the physicalserver P5 is coupled with the virtual server V2, transition of databetween the storage devices D does not occur and thus it is possible tohold allocation of the storage device D1 to the hash space. For thisreason, transition of data held in the storage device D due toreallocation to the hash space does not occur in the entirety of theinformation processing apparatus IPE1. The load of all of the operatingphysical servers P0 to P5 is reduced in a state where the load of anyone of the physical servers P0 to P5 is included at the low load region.For this reason, an increase in load due to replacing the physicalserver P5 with the virtual server V2 does not influence a serviceprovided by the information processing system SYS1. Accordingly, anincrease of the load of the information processing apparatus IPE1 byswitching the servers is minimized and thus it is possible to degradeprocessing performance of the information processing system SYS1 withoutan influence on a service provided by the information processing systemSYS1.

The control device CNTL may stop one of the physical servers P byperforming scale-in for performance in which the physical server P isreplaced with the virtual server V when the load of the physical serverP is small. With this, it is possible to decrease a processing capacityof the information processing apparatus IPE1 and to largely reduce powerconsumed by the information processing apparatus IPE1, compared to whenone of the physical servers P continues to be operated.

When the frequency of the load of any one of the physical servers P0 toP5 included at the low load region exceeds “0.8” in the state ST4(Condition D), the state of the information processing apparatus IPE1 ischanged to the state ST3 (the mark (m) in FIG. 5). An operation in thiscase is performed similarly to the change of the state ST5 to the stateST3. For example, configuration information (for example, a host nameand an IP address) of the physical server P5 and configurationinformation of the virtual server V2 are exchanged and the coupling ofthe storage device D5 is switched from the physical server P5 to thevirtual server V2. It is possible to obtain the same effect as theeffect obtained by changing the state ST5 to the state ST3, throughscale-in for performance in which the state ST4 is changed to the stateST3.

When the frequency of the load of any one of the physical servers P0 toP4 included at the low load region exceeds “0.8” in the state ST3(Condition E), the state of the information processing apparatus IPE1 ischanged to the state ST2 (the mark (n) in FIG. 5). In this case, datawhich is held in the storage device D5 coupled with the virtual serverV2 is transitioned to other storage devices D0 to D4 and the virtualserver V2 on the physical server P0 is terminated. That is, scale-in isperformed. When data transmitted from the user terminal TM is held induplicate in the plurality of storage devices D as replica data, thevirtual server V2 may be terminated ahead of transitioning data. In thiscase, the virtual server V2 is terminated and then data may betransitioned by using the replica data read from the storage device Dwhich is coupled with the physical servers P0 to P4 in operation or theother virtual server V in operation.

The load (Condition E) of the information processing apparatus IPE1 whenthe state ST3 is changed to the state ST2 is smaller than the load(Condition D) of the information processing apparatus IPE1 when thestate ST5 is changed to the state ST3. For this reason, when data istransitioned between the storage devices D depending on the change ofthe states of the information processing apparatus IPE1, it is alsopossible to reduce an influence on a service provided by the informationprocessing system SYS1 to the minimum. When Condition E is satisfied,one of the physical servers P may be stopped instead of terminating oneof the virtual servers V. Accordingly, it is possible to largely reducethe power consumed by the information processing apparatus IPE1,compared to when one of the virtual servers V is terminated.

When the frequency of the load of any one of the physical servers P0 toP4 included at the low load region exceeds “0.8” in the state ST2(Condition D), the state of the information processing apparatus IPE1 ischanged to the state ST1 (the mark (o) in FIG. 5). An operation in thiscase is performed similarly to the change of the state ST5 to the stateST3. For example, configuration information (for example, a host nameand an IP address) of the physical server P4 and configurationinformation of the virtual server V1 are exchanged and the coupling ofthe storage device D2 is switched from the physical server P4 to thevirtual server V1. That is, scale-in for performance is performed.

When the frequency of the load of any one of the physical servers P0 toP3 included at the low load region exceeds “0.8” in the state ST1(Condition D), the state of the information processing apparatus IPE1 ischanged to the state ST0 (the mark (p) in FIG. 5). Scale-in forperformance is performed as an operation performed in this case,similarly to the change of the state ST5 to the state ST3.

In FIG. 5, the control device CNTL replaces the virtual server V withthe physical server P when the frequency of the load of the physicalserver P included at the middle load region increases to exceed apredetermined value. That is, when the load of the informationprocessing apparatus IPE1 is small, the virtual server V is operated.When the load of the information processing apparatus IPE1 is large, thephysical server P is operated. When the load of the informationprocessing apparatus IPE1 is small, the virtual server V having accessefficiency (I/O performance) lower than that of the physical server P isoperated and thus it is possible to suppress an influence of the I/Operformance of the virtual server V on a service provided by theinformation processing system SYS1. When data transmitted from the userterminal TM is stored in duplicate in the plurality of storage devicesD, the physical server P is coupled with one of the plurality of storagedevices D storing the data and thus access efficiency of data isdetermined based on the I/O performance of the physical server P. Inthis case, it is possible to cover the I/O performance of the virtualserver V with the I/O performance of the physical server P.

FIG. 5 illustrates an example in which the control device CNTL changesthe state of the information processing apparatus IPE1 in accordancewith the frequency of the load of any one of the physical servers P.However, the control device CNTL may change the state of the informationprocessing apparatus IPE1 in accordance with a frequency of an averagevalue of the load of the operating physical server P. This is becausethe frequencies of the load of the respective operating physical serversP have a tendency to be similar to each other, as illustrated in FIG. 4.

FIG. 6 illustrates an example of the change of the load in an operationof the information processing system SYS1 illustrated in FIG. 5. Marksof (a) to (I) and (n) in FIG. 6 are the same as the marks indicatingholding or switching of the state of the information processingapparatus IPE1 illustrated in FIG. 5. FIG. 6 illustrates the frequencyof the load of any one of the physical servers P in operation. Marks ofCYC1 to CYC15 in FIG. 6 indicate a cycle of the control device CNTLcollecting the load information. In FIG. 6, for easily understandabledescription, the frequency of performing switching between the states STis higher than that in an actual operation of the information processingsystem SYS1.

First, in the state ST0, the load of the physical server P graduallybecomes higher and the frequency at the middle load region exceeds “0.8”in a cycle CYC3. Thus, the control device CNTL causes a state of theinformation processing apparatus IPE1 to be switched from the state ST0to the state ST1 ((g) in FIG. 6). Since switching of the state ST0 tothe state ST1 causes the virtual server V0 to be replaced with thephysical server P3, the processing capacity of the informationprocessing apparatus IPE1 is improved and the frequency at the middleload region decreases in a cycle CYC4.

Then, in the state ST1, the load of the physical server P graduallybecomes higher and the frequency at the middle load region exceeds “0.8”in a cycle CYC5. Thus, the control device CNTL causes a state of theinformation processing apparatus IPE1 to be switched from the state ST1to the state ST2 ((h) in FIG. 6). Since switching of the state ST1 tothe state ST2 causes the virtual server V1 to be replaced with thephysical server P4, the processing capacity of the informationprocessing apparatus IPE1 is improved and the frequency at the middleload region decreases in a cycle CYC6.

In the state ST2, the load of the physical server P becomes higher andthe frequency at the high load region exceeds “0.9” in a cycle CYC7 (1).Thus, the control device CNTL causes a state of the informationprocessing apparatus IPE1 to be switched from the state ST2 to the stateST4 ((j) in FIG. 6). Since switching of the state ST2 to the state ST4causes the new physical server P5 to be operated, the processingcapacity of the information processing apparatus IPE1 is improved andthe frequency at the high load region decreases in a cycle CYC8.

In the state ST2, the load of the physical server P gradually becomeshigher and the frequency at the middle load region exceeds “0.7” in thecycle CYC5. Thus, the control device CNTL causes a state of theinformation processing apparatus IPE1 to be switched from the state ST2to the state ST3 ((i) in FIG. 6). Since switching of the state ST2 tothe state ST3 causes the new virtual server V2 to be operated, theprocessing capacity of the information processing apparatus IPE1 isimproved and the frequency at the middle load region decreases in acycle CYC9.

In the state ST3, the load of the physical server P gradually becomeshigher and the frequency at the middle load region exceeds “0.8” in acycle CYC10. Thus, the control device CNTL causes a state of theinformation processing apparatus IPE1 to be switched from the state ST3to the state ST5 ((k) in FIG. 6). Since switching of the state ST3 tothe state ST5 causes the virtual server V2 to be replaced with thephysical server P5, the processing capacity of the informationprocessing apparatus IPE1 is improved and the frequency at the middleload region decreases in a cycle CYC11.

In the state ST5, the load of the physical server P becomes lower andthe frequency at the low load region exceeds “0.8” in a cycle CYC12.Thus, the control device CNTL causes a state of the informationprocessing apparatus IPE1 to be switched from the state ST5 to the stateST3 ((I) in FIG. 6). Since switching of the state ST5 to the state ST3causes the physical server P5 to be replaced with the virtual server V2,the processing capacity of the information processing apparatus IPE1decreases and the frequency at the middle load region increases in acycle CYC13.

In the state ST3, the load of the physical server P becomes furtherlower and the frequency at the low load region exceeds “0.9” in a cycleCYC14. Thus, the control device CNTL causes a state of the informationprocessing apparatus IPE1 to be switched from the state ST3 to the stateST2 ((n) in FIG. 6). Since switching of the state ST3 to the state ST2causes the virtual server V2 to be terminated, the processing capacityof the information processing apparatus IPE1 decreases and the frequencyat the middle load region increases in a cycle CYC15.

FIG. 7 illustrates another example of a change of the load in anoperation of the information processing system SYS1 illustrated in FIG.5. Detailed description of operations the same as or similar to those inFIG. 6 will be omitted. Marks of (c) to (e) and (m) to (o) in FIG. 7 arethe same as the marks indicating holding or switching of the state ofthe information processing apparatus IPE1 illustrated in FIG. 5. Forexample, FIG. 7 illustrates operations subsequent to the cycle CYC8 inFIG. 6. The change of the state ST4 to the state ST3 and the change ofthe state ST2 to the state ST1 are similar to the change of the stateST5 to the state ST3.

FIG. 8 illustrates an example of an operating flow in the control deviceCNTL illustrated in FIG. 1. For example, the operating flow illustratedin FIG. 8 is implemented by the control device CNTL executing thecontrol program of the information processing system SYS1. The controldevice CNTL is started when the information processing system SYS1starts a function as a storage server or a data center.

First, in Step S10, the control device CNTL initializes the variablesHUTL, MUTL, and LUTL to be “0”. In Step S12, the control device CNTLdetermines whether or not the information processing apparatus IPE1 isin the state ST0. When the information processing apparatus IPE1 is inthe state ST0, the process proceeds to an ST0 control routine of StepS100. When the information processing apparatus IPE1 is not in the stateST0, the process proceeds to Step S14. In the ST0 control routine, it isdetermined whether a state of the information processing apparatus IPE1is transitioned from the state ST0 to the state ST1 or is held to be inthe state ST0. FIG. 9 illustrates an example of the ST0 control routineand FIG. 15 illustrates an example of a program of the ST0 controlroutine.

In Step S14, the control device CNTL determines whether or not theinformation processing apparatus IPE1 is in the state ST1. When theinformation processing apparatus IPE1 is in the state ST1, the processproceeds to an ST1 control routine of Step S200. When the informationprocessing apparatus IPE1 is not in the state ST1, the process proceedsto Step S16. In the ST1 control routine, it is determined whether astate of the information processing apparatus IPE1 is transitioned fromthe state ST1 to the state ST2, is transitioned from the state ST1 tothe state ST0, or is held to be in the state ST1. FIG. 10 illustrates anexample of a flow of the ST1 control routine and FIG. 16 illustrates anexample of a program of the ST1 control routine.

In Step S16, the control device CNTL determines whether or not theinformation processing apparatus IPE1 is in the state ST2. When theinformation processing apparatus IPE1 is in the state ST2, the processproceeds to an ST2 control routine of Step S300. When the informationprocessing apparatus IPE1 is not in the state ST2, the process proceedsto Step S18. In the ST2 control routine, it is determined whether astate of the information processing apparatus IPE1 is transitioned fromthe state ST2 to the state ST4, is transitioned from the state ST2 tothe state ST3, is transitioned from the state ST2 to the state ST1, oris held to be in the state ST2. FIG. 11 illustrates an example of a flowof the ST2 control routine and FIG. 17 illustrates an example of aprogram of the ST2 control routine.

In Step S18, the control device CNTL determines whether or not theinformation processing apparatus IPE1 is in the state ST3. When theinformation processing apparatus IPE1 is in the state ST3, the processproceeds to an ST3 control routine of Step S400. When the informationprocessing apparatus IPE1 is not in the state ST3, the process proceedsto Step S20. In the ST3 control routine, it is determined whether astate of the information processing apparatus IPE1 is transitioned fromthe state ST3 to the state ST5, is transitioned from the state ST3 tothe state ST2, or is held to be in the state ST3. FIG. 12 illustrates anexample of a flow of the ST3 control routine and FIG. 18 illustrates anexample of a program of the ST3 control routine.

In Step S20, the control device CNTL determines whether or not theinformation processing apparatus IPE1 is in the state ST4. When theinformation processing apparatus IPE1 is in the state ST4, the processproceeds to an ST4 control routine of Step S500. When the informationprocessing apparatus IPE1 is not in the state ST4, the process proceedsto Step S22. In the ST4 control routine, it is determined whether astate of the information processing apparatus IPE1 is transitioned fromthe state ST4 to the state ST3 or is held to be in the state ST4. FIG.13 illustrates an example of a flow of the ST4 control routine and FIG.19 illustrates an example of a program of the ST4 control routine.

In Step S22, the control device CNTL determines whether or not theinformation processing apparatus IPE1 is in the state ST5. When theinformation processing apparatus IPE1 is in the state ST5, the processproceeds to an ST5 control routine of Step S600. When the informationprocessing apparatus IPE1 is not in the state ST5, the process returnsto Step S10. In the ST5 control routine, it is determined whether astate of the information processing apparatus IPE1 is transitioned fromthe state ST5 to the state ST3 or is held to be in the state ST5. FIG.14 illustrates an example of a flow of the ST5 control routine and FIG.20 illustrates an example of a program of the ST5 control routine. Thecontrol device CNTL executes Steps S100, S200, S300, S400, S500, andS600 and then causes the process to return to Step S10.

FIG. 9 illustrates an example of the ST0 control routine illustrated inFIG. 8. First, in Step S102, the control device CNTL resets the timecount TC to “0”. In Step S104, the control device CNTL determineswhether or not the load of any one of the physical servers P exceeds80%. When the load exceeds 80%, the process proceeds to Step S106. Whenthe load is equal to or less than 80%, the process proceeds to StepS108. In Step S106, the control device CNTL increases the variable HUTLby “one” and causes the process to proceed to Step S116.

In Step S108, the control device CNTL determines whether or not the loadof any one of the physical servers P exceeds 30%. When the load exceeds30%, that is, when the load exceeds 30% and is equal to or less than80%, the process proceeds to Step S110. When the load is equal to orless than 30%, the process proceeds to Step S112. In Step S110, thecontrol device CNTL increases the variable MUTL by “one” and causes theprocess to proceed to Step S116.

In Step S112, the control device CNTL determines whether or not the loadof any one of the physical servers P is less than 2%. When the load isless than 2%, the process proceeds to Step S114. When the load is equalto or more than 2%, that is, when the load is 2% or more and 30% orless, the process proceeds to Step S116. In Step S114, the controldevice CNTL increases the variable LUTL by “one” and causes the processto proceed to Step S116.

In Step S116, the control device CNTL waits for, for example, one secondand then causes the process to proceed to Step S118. In Step S118, thecontrol device CNTL increases the time count TC by “one”. In Step S120,the control device CNTL determines whether or not the time count TCexceeds “600”. When the time count TC exceeds “600”, that is, when anaggregate time of the variables HUTL, MUTL, and LUTL exceeds 600seconds, the process proceeds to Step S122. When the time count TC isequal to or less than “600”, the process returns to Step S104.

In Step S122, the control device CNTL determines whether or not thefrequency of the load at the middle load region, which is a valueobtained by dividing the value of the variable MUTL by the time count TCexceeds “0.8”. When the frequency of the load at the middle load regionexceeds “0.8”, the process proceeds to Step S124. When the frequency ofthe load at the middle load region is equal to or less than “0.8”, theprocess proceeds to Step S126.

In Step S124, the control device CNTL causes a state of the informationprocessing apparatus IPE1 to be transitioned from the state ST0 to thestate ST1. In Step S126, the control device CNTL causes the informationprocessing apparatus IPE1 to be held in the state ST0. After theprocesses of Steps S124 and S126, the ST0 control routine is ended. TheST0 control routine is executed for each physical server P in operation.When any one of the physical servers P satisfies a determinationcondition in Step S122 (Yes in S122), Step S124 is processed.

FIG. 10 illustrates an example of the ST1 control routine illustrated inFIG. 8. Detailed description of processes the same as or similar tothose in FIG. 9 will be omitted. Processes from Step S102 to Step S120are the same as those in FIG. 9.

In Step S222, the control device CNTL performs the same determination asthat in Step S122 illustrated in FIG. 9. When the frequency of the loadat the middle load region exceeds “0.8”, the control device CNTL causesa state of the information processing apparatus IPE1 to be transitionedfrom the state ST1 to the state ST2 in Step S224. When the frequency ofthe load at the middle load region is equal to or less than “0.8”, theprocess proceeds to Step S226.

In Step S226, the control device CNTL determines whether or not thefrequency of the load at the low load region, which is a value obtainedby dividing the value of the variable LUTL by the time count TC exceeds“0.8”. When the frequency of the load at the low load region exceeds“0.8”, the process proceeds to Step S228. When the frequency of the loadat the low load region is equal to or less than “0.8”, the processproceeds to Step S230. In Step S228, the control device CNTL causes astate of the information processing apparatus IPE1 to be transitionedfrom the state ST1 to the state ST0. In Step S230, the control deviceCNTL causes the information processing apparatus IPE1 to be held in thestate ST1. After processes of Steps S224, S228, and S230, the ST1control routine is ended.

The ST1 control routine is executed for each physical server P inoperation. When any one of the physical servers P satisfies adetermination condition in Step S222 (Yes in S222), Step S224 isprocessed. When any one of the physical servers P satisfies adetermination condition in Step S226 (Yes in S226), Step S228 isprocessed. The summation of frequency MUTL/TC and frequency LUTL/TC doesnot exceed 1.0. Thus, the determination conditions of Steps S222 andS226 are not simultaneously satisfied. As illustrated in FIG. 4, theload of the plurality of physical servers P which are operating ischanged with a similar tendency. For this reason, it does not occur thatone of the plurality of physical servers P satisfies the determinationcondition of Step S222 and another of the plurality of physical serversP satisfies the determination condition of Step S226.

FIG. 11 illustrates an example of the ST2 control routine illustrated inFIG. 8. Detailed description of processes the same as or similar tothose in FIGS. 9 and 10 will be omitted. The processes from Step S102 toStep S120 are the same as those in FIG. 9.

In Step S322, the control device CNTL determines whether or not thefrequency of the load at the high load region, which is a value obtainedby dividing the value of the variable HUTL by the time count TC exceeds“0.9”. When the frequency of the load at the high load region exceeds“0.9”, the process proceeds to Step S324. When the frequency of the loadat the high load region is equal to or less than “0.9”, the processproceeds to Step S326. In Step S324, the control device CNTL causes astate of the information processing apparatus IPE1 to be transitionedfrom the state ST2 to the state ST4.

In Step S326, the control device CNTL determines whether or not thefrequency of the load at the middle load region, which is a valueobtained by dividing the value of the variable MUTL by the time count TCexceeds “0.7”. When the frequency of the load at the middle load regionexceeds “0.7”, the process proceeds to Step S328. When the frequency ofthe load at the middle load region is equal to or less than “0.7”, theprocess proceeds to Step S330. In Step S328, the control device CNTLcauses a state of the information processing apparatus IPE1 to betransitioned from the state ST2 to the state ST3.

In Step S330, the control device CNTL performs determination similar tothat in Step S226 illustrated in FIG. 10. When the frequency of the loadat the low load region exceeds “0.8”, the control device CNTL causes astate of the information processing apparatus IPE1 to be transitionedfrom the state ST2 to the state ST1 in Step S332. When the frequency ofthe load at the low load region is equal to or less than “0.8”, theprocess proceeds to Step S334. In Step S334, the control device CNTLcauses the information processing apparatus IPE1 to be held in the stateST2. After the processes of Steps S324, S328, S332, and S334, the ST2control routine is ended.

The ST2 control routine is executed for each physical server P inoperation. When any one of the physical servers P satisfies adetermination condition in Step S322 (Yes in S322), Step S324 isprocessed. When any one of the physical servers P satisfies adetermination condition in Step S326 (Yes in S326), Step S328 isprocessed. When any one of the physical servers P satisfies adetermination condition in Step S330 (Yes in S330), Step S334 isprocessed. Similarly to FIG. 10, the summation of frequency HUTL/TC,frequency MUTL/TC, and frequency LUTL/TC does not exceed 1.0. Thus, thedetermination conditions of Steps S322, S326, and S330 are notsimultaneously satisfied. Similarly to FIG. 10, it does not occur thatthe plurality of physical servers P which are operating satisfy thedetermination conditions (S322, S326, and S330) different from eachother.

FIG. 12 illustrates an example of the ST3 control routine illustrated inFIG. 8. Detailed description of processes the same as or similar tothose in FIGS. 9 and 10 will be omitted. The processes from Step S102 toStep S120 are the same as those in FIG. 9.

In Step S422, the control device CNTL performs the same determination asthat in Step S122 illustrated in FIG. 9. When the frequency of the loadat the middle load region exceeds “0.8”, the control device CNTL causesa state of the information processing apparatus IPE1 to be transitionedfrom the state ST3 to the state ST5 in Step S424. When the frequency ofthe load at the middle load region is equal to or less than “0.8”, theprocess proceeds to Step S426.

In Step S426, the control device CNTL performs the same determination asthat in Step S226 illustrated in FIG. 10. When the frequency of the loadat the low load region exceeds “0.9”, the control device CNTL causes astate of the information processing apparatus IPE1 to be transitionedfrom the state ST3 to the state ST2 in Step S428. When the frequency ofthe load at the low load region is equal to or less than “0.9”, theprocess proceeds to Step S430. In Step S430, the control device CNTLcauses the information processing apparatus IPE1 to be held in the stateST3. After processes of Steps S424, S428, and S430, the ST3 controlroutine is ended.

The ST3 control routine is executed for each physical server P inoperation. When any one of the physical servers P satisfies adetermination condition in Step S422 (Yes in S422), Step S424 isprocessed. When any one of the physical servers P satisfies adetermination condition in Step S426 (Yes in S426), Step S428 isprocessed. The summation of frequency MUTL/TC and frequency LUTL/TC doesnot exceed 1.0. Thus, the determination conditions of Steps S422 andS426 are not simultaneously satisfied. Similarly to FIG. 10, it does notoccur that the plurality of physical servers P which are operatingsatisfy the determination conditions (S422 and S426) different from eachother.

FIG. 13 illustrates an example of the ST4 control routine illustrated inFIG. 8. Detailed description of processes the same as or similar tothose in FIGS. 9 and 10 will be omitted. The processes from Step S102 toStep S120 are the same as those in FIG. 9.

In Step S522, the control device CNTL performs the same determination asthat in Step S226 illustrated in FIG. 10. When the frequency of the loadat the low load region exceeds “0.8”, the control device CNTL causes astate of the information processing apparatus IPE1 to be transitionedfrom the state ST4 to the state ST3 in Step S524. When the frequency ofthe load at the low load region is equal to or less than “0.8”, theprocess proceeds to Step S526. In Step S526, the control device CNTLcauses the information processing apparatus IPE1 to be held in the stateST4. After processes of Steps S524 and S526, the ST4 control routine isended. The ST4 control routine is executed for each physical server P inoperation. When any one of the physical servers P satisfies adetermination condition in Step S522 (Yes in S522), Step S524 isprocessed.

FIG. 14 illustrates an example of the ST5 control routine illustrated inFIG. 8. Detailed description of processes the same as or similar tothose in FIGS. 9 and 10 will be omitted. The processes from Step S102 toStep S120 are the same as those in FIG. 9.

In Step S622, the control device CNTL performs the same determination asthat in Step S226 illustrated in FIG. 10. When the frequency of the loadat the low load region exceeds “0.8”, the control device CNTL causes astate of the information processing apparatus IPE1 to be transitionedfrom the state ST5 to the state ST3 in Step S624. When the frequency ofthe load at the low load region is equal to or less than “0.8”, theprocess proceeds to Step S626. In Step S626, the control device CNTLcauses the information processing apparatus IPE1 to be held in the stateST5. After processes of Steps S624 and S626, the ST5 control routine isended. The ST5 control routine is executed for each physical server P inoperation. When any one of the physical servers P satisfies adetermination condition in Step S622 (Yes in S622), Step S624 isprocessed.

FIG. 15 illustrates an example of a program of executing the ST0 controlroutine illustrated in FIG. 9. For example, FIG. 15 illustrates a sourcelist of a program executed by the CPU of the control device CNTL and arow number is added to the forefront in each row, for convenientdescription, in FIG. 15. Characters subsequent to a mark of “//”indicate a comment.

Variables are defined by using “DEFINDE” from a first row to a ninthrow. A variable HTH indicates the threshold value VT1 (80%) illustratedin FIG. 3. A variable MTH indicates the threshold value VT2 (30%)illustrated in FIG. 3. A variable LTH indicates the threshold value VT3(2%) illustrated in FIG. 3. A variable TIME indicates the time count TCillustrated in FIG. 3 and a collecting period (for example, 600 seconds)for collecting the load information of the physical server P.

A variable RPEXP indicates the frequency of the load (0.9; thresholdvalue) at the high load region, for determining addition of the physicalserver P. A variable RVEXP indicates the frequency of the load (0.8;threshold value) at the middle load region, for determining replacementof the virtual server V with the physical server P. A variable RVNEWindicates the frequency of the load (0.7; threshold value) at the middleload region, for determining addition of the virtual server V. Avariable RSH indicates the frequency of the load (0.8; threshold value)at the low load region, for determining a termination of the virtualserver V. A variable RVRM indicates the frequency of the load (0.9;threshold value) at the low load region, for determining replacement ofthe physical server P with the virtual server V.

When the state (State) of the information processing apparatus IPE1 isST0 (=0) in an 11th row, rows from a 12th row to a 23rd row areexecuted. A variable measureCpuUtil in a 13th row, a 15th row, and a17th row indicates the load of the physical server P for each time countTC, which is held in the holding section HLD. “do transState” in a 21strow and a 23rd row refers to a command for changing the state of theinformation processing apparatus IPE1. The comments added to therespective rows indicate corresponding of processes executed in the rowsfrom the 12th row to the 23rd row to Steps illustrated in FIG. 9.

FIG. 16 illustrates an example of a program of executing the ST1 controlroutine illustrated in FIG. 10. Detailed description regardingstatements the same as or similar to those in FIG. 15 will be omitted.Definitions of variables by “DEFINDE” in rows from a first row to aninth row are the same as those in FIG. 15. When the state (State) ofthe information processing apparatus IPE1 is ST1 (=1) in an 11th row,rows from a 12th row to a 25th row are executed. Comments added to therespective rows indicate corresponding of processes executed in the rowsfrom the 12th row to the 25th row to Steps illustrated in FIG. 10.

FIG. 17 illustrates an example of a program of executing the ST2 controlroutine illustrated in FIG. 11. Detailed description regardingstatements the same as or similar to those in FIG. 15 will be omitted.Definitions of variables by “DEFINDE” in rows from a first row to aninth row are the same as those in FIG. 15. When the state (State) ofthe information processing apparatus IPE1 is ST2 (=2) in an 11th row,rows from a 12th row to a 27th row are executed. Comments added to therespective rows indicate corresponding of processes executed in the rowsfrom the 12th row to the 27th row to Steps illustrated in FIG. 11.

FIG. 18 illustrates an example of a program of executing the ST3 controlroutine illustrated in FIG. 12. Detailed description regardingstatements the same as or similar to those in FIG. 15 will be omitted.Definitions of variables by “DEFINDE” in rows from a first row to aninth row are the same as those in FIG. 15. When the state (State) ofthe information processing apparatus IPE1 is ST3 (=3) in an 11th row,rows from a 12th row to a 25th row are executed. Comments added to therespective rows indicate corresponding of processes executed in the rowsfrom the 12th row to the 25th row to Steps illustrated in FIG. 12.

FIG. 19 illustrates an example of a program of executing the ST4 controlroutine illustrated in FIG. 13. Detailed description regardingstatements the same as or similar to those in FIG. 15 will be omitted.Definitions of variables by “DEFINDE” in rows from a first row to aninth row are the same as those in FIG. 15. When the state (State) ofthe information processing apparatus IPE1 is ST4 (=4) in an 11th row,rows from a 12th row to a 23rd row are executed. Comments added to therespective rows indicate corresponding of processes executed in the rowsfrom the 12th row to the 23rd row to Steps illustrated in FIG. 13.

FIG. 20 illustrates an example of a program of executing the ST5 controlroutine illustrated in FIG. 14. Detailed description regardingstatements the same as or similar to those in FIG. 15 will be omitted.Definitions of variables by “DEFINDE” in rows from a first row to aninth row are the same as those in FIG. 15. When the state (State) ofthe information processing apparatus IPE1 is ST5 (=5) in an 11th row,rows from a 12th row to a 23rd row are executed. Comments added to therespective rows indicate corresponding of processes executed in the rowsfrom the 12th row to the 23rd row to Steps illustrated in FIG. 14.

In the above description, in the embodiment illustrated in FIGS. 1 to20, it is possible to change the processing performance of theinformation processing system SYS1 depending on a change of the load ofthe physical server P without an influence of the information processingsystem SYS1 on efficiency in information processing. For example, thecontrol device CNTL detects that the load of the physical server P tendsto increase and replaces the virtual server V with the physical server Pand thus it is possible to increase the processing capacity of theinformation processing apparatus IPE1. At this time, switching thecoupling of the storage device D from the virtual server V to thephysical server P causes data not to be transitioned between the storagedevices D. Accordingly, it is possible to improve the processingperformance of the information processing system SYS1 without aninfluence on a service provided by the information processing systemSYS1.

Addition of the virtual server V resulting in transition of data isperformed during a time period when a tendency of increasing the load ofthe physical server P is lower than a tendency of increasing the loadwhen the virtual server V is replaced with the physical server P. Thus,it is possible to reduce an influence of data transition processingwhich occurs due to addition of the virtual server V, on a service ofthe storage server and the like, compared to when the load becomes largeand thus data is transitioned.

When the load of the physical server P becomes small, it is possible tostop one of the physical servers P by replacing the physical server Pwith the virtual server V. Accordingly, it is possible to decrease theprocessing capacity of the information processing apparatus IPE1 and tolargely reduce the power consumed by the information processingapparatus IPE1, compared to when one of the physical servers P continuesto be in operation. At this time, switching the coupling of the storagedevice D from the physical server P to the virtual server V causes datanot to be transitioned between the storage devices D. Accordingly, it ispossible to degrade the processing performance of the informationprocessing system SYS1 without an influence on a service provided by theinformation processing system SYS1.

Removal of the virtual server V resulting in transition of data isperformed when the load of the physical server P is smaller than a loadwhen the physical server P is replaced with the virtual server V.Accordingly, it is also possible to reduce an influence on a serviceprovided by the information processing system SYS1 to the minimum whenremoving the virtual server V causes data to be transitioned between thestorage devices D.

FIG. 21 illustrates an example in which a control device changes thestate of an information processing apparatus IPE1 in another embodimentof the information processing system, the control method of theinformation processing system, and the control program of theinformation processing system. A control device CNTL which executesprocesses illustrated in FIG. 21 and an information processing apparatusIPE1 which is controlled by the control device CNTL are similar to thosein FIG. 1 except that a program executed by the control device CNTL isdifferent. FIG. 21 illustrates a control method executed by the controldevice CNTL mounted in the information processing system SYS1illustrated in FIG. 1 and processes by the control program executed bythe control device CNTL. Detailed descriptions of components the same asor similar to those in the embodiment illustrated in FIGS. 1 to 20 willbe omitted.

“NOW” illustrated in FIG. 21 indicates the current state of theinformation processing apparatus IPE1. Processes A to E illustratedaround the state “NOW” in FIG. 21 represent operations of theinformation processing apparatus IPE1 executed by control of the controldevice CNTL. Dot-lined arrows indicate that the state of the informationprocessing apparatus IPE1 which executes processes by control of thecontrol device CNTL becomes the state “NOW” anew. In FIG. 21, upwardarrows which are positioned behind numerical values of the load and thefrequency indicate that the load and the frequency are larger than thenumerical values. Each downward arrow which is positioned behind anumerical value of the load indicates that the load is smaller than eachof the numerical values. In FIG. 21, “load” and “frequency” are the sameas “load” and “frequency” which are described with reference to FIGS. 1to 20.

For example, when the frequency of that the load of any one of thephysical servers P in operation exceeds 30% and is equal to or less than80% exceeds “0.8” ((Condition A) described with reference to FIG. 3),the control device CNTL executes Process A of changing the state of theinformation processing apparatus IPE1. The control device CNTL replacesone of the virtual servers V with one of the physical servers P byterminating any one of the virtual servers V and operating any one ofthe physical servers P, in Process A. The storage device D which hasbeen coupled with the virtual server V to be terminated is coupled withthe physical server P to be operated.

For example, the change of the state of the information processingapparatus IPE1 through Process A is the same as the change of the stateST0 to the state ST1, the change of the state ST1 to the state ST2, andthe change of the state ST3 to the state ST5 which are illustrated inFIG. 5. When the load exceeds 80%, Process C may be executed prior toProcess A in accordance with “frequency”.

When the frequency of the load of any one of the physical servers P inoperation exceeds 30% and is equal to or less than 80% exceeds “0.7”((Condition B) described with reference to FIG. 3), the control deviceCNTL executes Process B of changing the state of the informationprocessing apparatus IPE1. The control device CNTL adds one of thevirtual servers V to the information processing apparatus IPE1 bycausing any one of the terminated virtual servers V to be operated onany one of the physical servers P in operation, in Process B. Forexample, the change of the state of the information processing apparatusIPE1 through Process B is the same as the change of the state ST2 to thestate ST3 illustrated in FIG. 5. When the frequency exceeds “0.8”,Process A is executed prior to Process B. When the load exceeds 80%,Process C may be executed prior to Process B in accordance with“frequency”.

When the frequency of the load of any one of the physical servers P inoperation exceeding 80% exceeds “0.9” ((Condition C) described withreference to FIG. 3), the control device CNTL executes Process C ofchanging the state of the information processing apparatus IPE1. Thecontrol device CNTL adds one of the physical servers P to theinformation processing apparatus IPE1 by causing any one of thecurrently stopped physical servers P to be operated, in Process C. Forexample, change of the state of the information processing apparatusIPE1 through Process C is the same as change of the state ST2 to thestate ST4 illustrated in FIG. 5.

When the frequency of the load of any one of the physical servers P inoperation at less than 2% exceeds “0.8” ((Condition D) described withreference to FIG. 3), the control device CNTL executes Process D ofchanging the state of the information processing apparatus IPE1. Thecontrol device CNTL replaces one of the physical servers P with one ofthe virtual servers V by causing any one of the physical servers P to bestopped and causing any one of the virtual servers V to be operated, inProcess D. The storage device D which has been coupled with the physicalserver P to be stopped is coupled with the virtual server V to beoperated anew.

For example, the change of the state of the information processingapparatus IPE1 through Process D is the same as the change of the stateST1 to the state ST0, the change of the state ST2 to the state ST1, thechange of the state ST4 to the state ST3, and the change of the stateST5 to the state ST3 which are illustrated in FIG. 5. When the frequencyexceeds “0.9”, Process E is executed prior to Process D.

When the frequency of the load of any one of the physical servers P inoperation at less than 2% exceeds “0.9” ((Condition E) described withreference to FIG. 3), the control device CNTL executes Process E ofchanging the state of the information processing apparatus IPE1. Thecontrol device CNTL removes one of the virtual servers V from theinformation processing apparatus IPE1 by terminating any one of theoperating virtual servers V, in Process E. For example, the change ofthe state of the information processing apparatus IPE1 through Process Eis the same as the change of the state ST3 to the state ST2 illustratedin FIG. 5.

When the load of any one of the physical servers P in operationsatisfies no condition for executing Process A, Process B, Process C,Process D, and Process E, the control device CNTL holds the state of theinformation processing apparatus IPE1 without changing (“F” in FIG. 21).

FIG. 22 illustrates an example of a relationship between each of ProcessA, Process B, Process C, Process D, and Process E, the load, and thefrequency illustrated in FIG. 21. As described with reference to FIG.21, Process C has the primary priority and Process A has the secondarypriority in a condition obtained by overlapping the conditions forexecuting Process A, Process B, and Process C with each other. Process Ehas a priority in a condition obtained by overlapping the conditions forexecuting Process D and Process E with each other.

In the above description, in the embodiment illustrated in FIGS. 21 and22, it is also possible to obtain an effect similar to the effectobtained in the embodiment illustrated in FIGS. 1 to 20. That is, it ispossible to improve the processing performance of the informationprocessing system SYS1 depending on the change of the load of thephysical server P without an influence of the information processingsystem SYS1 on efficiency in information processing. Additionally, it ispossible to degrade the processing performance of the informationprocessing system SYS1 depending on the change of the load of thephysical server P without an influence of the information processingsystem SYS1 on efficiency in information processing.

Features and advantages according to the embodiments will be apparentfrom the above-described detailed description. This intends to have thefeatures and the advantages according to the embodiments withoutdeparting from the spirit and scope of the claims. A person withordinary skill in the art will easily consider various enhancements andchanges. Accordingly, appropriate improvements and substitutions whichcould be included in the scope disclosed in the embodiments may be madewithout the intention to limit the range of the embodiments having theinventive characteristics according to the above description.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the inventionand the concepts contributed by the inventor to furthering the art, andare to be construed as being without limitation to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although the embodiments of the presentinvention have been described in detail, it should be understood thatthe various changes, substitutions, and alterations could be made heretowithout departing from the spirit and scope of the invention.

What is claimed is:
 1. An information processing system comprising: aplurality of processors configured to cause a virtual machine on eachprocessor to be operated or terminated; a plurality of storage devices;and a control device configured to: collect processing loads of therespective processors, cause a first virtual machine which operates on afirst processor in which a frequency of a processing load exceeding afirst threshold exceeds a first proportion, to be terminated, cause asecond processor which is stopped among the plurality of processors, tobe operated, and cause a first storage device which is coupled with theterminated first virtual machine among the plurality of storage devices,to be coupled with the operated second processor.
 2. The informationprocessing system according to claim 1, wherein the control device isconfigured to: when a frequency of a processing load of a thirdprocessor among the plurality of processors exceeding the firstthreshold exceeds a second proportion smaller than the first proportion,cause a terminated second virtual machine to be operated on a fourthprocessor which is operating among the plurality of processors, andcause the operated second virtual machine to be coupled with a secondstorage device among the plurality of storage devices, which is notcoupled with the plurality of processors and the virtual machines. 3.The information processing system according to claim 1, wherein thecontrol device is configured to: when a frequency of a processing loadof a fifth processor among the plurality of processors exceeding asecond threshold higher than the first threshold exceeds a thirdproportion higher than the first proportion, cause a sixth processorwhich is stopped among the plurality of processors, to be operated, andcause the sixth processor to be coupled with a third storage devicewhich is not coupled with the plurality of processors and the virtualmachines among the plurality of storage devices.
 4. The informationprocessing system according to claim 1, wherein the control device isconfigured to: when a frequency of a processing load of a seventhprocessor among the plurality of processors being less than a thirdthreshold lower than the first threshold exceeds a fourth proportion,cause an eighth processor which does not operate the virtual machineamong the plurality of processors, to be stopped, cause a third virtualmachine to be operated on a ninth processor which is operating among theplurality of processors, and cause the first storage device to becoupled with the operated third virtual machine.
 5. The informationprocessing system according to claim 4, wherein the control device isconfigured to: when a frequency of a processing load of a tenthprocessor among the plurality of processors being less than the thirdthreshold exceeds a fifth proportion higher than the fourth proportion,cause data which is held in a fourth storage device coupled with afourth virtual machine which is operating on an eleventh processor amongthe plurality of processors to be transmitted to a fifth storage devicewhich is coupled with an operating fifth virtual machine among theplurality of virtual machine or coupled with an operating twelfthprocessor among the plurality of processors, cause the fourth virtualmachine to be terminated, and separate the fourth storage device fromthe fourth virtual machine.
 6. The information processing systemaccording to claim 1, further comprising: a switch configured to coupleany one of the plurality of processors and the virtual machines on theplurality of processors with any one of the plurality of storagedevices.
 7. A method of controlling an information processing systemincluding a plurality of processors and a plurality of storage devices,the plurality of processors causing a virtual machine on each processorto be operated or terminated, the method comprising: collectingprocessing loads of the respective processors; terminating a firstvirtual machine which operates on a first processor in which a frequencyof a processing load exceeding a first threshold exceeds a firstproportion; operating a second processor which is stopped among theplurality of processors; and coupling, by a processor, a first storagedevice which is coupled with the terminated first virtual machine amongthe plurality of storage devices, with the operated second processor. 8.The method according to claim 7, further comprising: when a frequency ofa processing load of a third processor among the plurality of processorsexceeding the first threshold exceeds a second proportion smaller thanthe first proportion, operating a terminated second virtual machine on afourth processor which is operating among the plurality of processors;and coupling the operated second virtual machine with a second storagedevice among the plurality of storage devices, which is not coupled withthe plurality of processors and the virtual machines.
 9. The methodaccording to claim 7, further comprising: when a frequency of aprocessing load of a fifth processor among the plurality of processorsexceeding a second threshold higher than the first threshold exceeds athird proportion higher than the first proportion, operating a sixthprocessor which is stopped among the plurality of processors; andcoupling the sixth processor with a third storage device which is notcoupled with the plurality of processors and the virtual machines amongthe plurality of storage devices.
 10. The method according to claim 7,further comprising: when a frequency of a processing load of a seventhprocessor among the plurality of processors being less than a thirdthreshold lower than the first threshold exceeds a fourth proportion,stopping an eighth processor which does not operate the virtual machineamong the plurality of processors; operating a third virtual machine ona ninth processor which is operating among the plurality of processors;and coupling the first storage device with the operated third virtualmachine.
 11. The method according to claim 10, further comprising: whena frequency of a processing load of a tenth processor among theplurality of processors being less than the third threshold exceeds afifth proportion higher than the fourth proportion, transmitting datawhich is held in a fourth storage device coupled with a fourth virtualmachine which is operating on an eleventh processor among the pluralityof processors to a fifth storage device which is coupled with anoperating fifth virtual machine among the plurality of virtual machineor coupled with an operating twelfth processor among the plurality ofprocessors; terminating the fourth virtual machine; and separating thefourth storage device from the fourth virtual machine.
 12. Anon-transitory computer-readable storage medium storing a program thatcauses a computer to execute a process, the process comprising:collecting processing loads of a plurality of processors configuring tocause a virtual machine on each processor to be operated or terminated;causing a first virtual machine which operates on a first processor inwhich a frequency of a processing load exceeding a first thresholdexceeds a first proportion, to be terminated; causing a second processorwhich is stopped among the plurality of processors, to be operated; andcausing a first storage device which is coupled with the terminatedfirst virtual machine among a plurality of storage devices, to becoupled with the operated second processor.
 13. The non-transitorycomputer-readable storage medium according to claim 12, wherein theprocess further comprising: when a frequency of a processing load of athird processor among the plurality of processors exceeding the firstthreshold exceeds a second proportion smaller than the first proportion,causing a terminated second virtual machine to be operated on a fourthprocessor which is operating among the plurality of processors; andcausing the operated second virtual machine to be coupled with a secondstorage device among the plurality of storage devices, which is notcoupled with the plurality of processors and the virtual machines. 14.The non-transitory computer-readable storage medium according to claim12, wherein the process further comprising: when a frequency of aprocessing load of a fifth processor among the plurality of processorsexceeding a second threshold higher than the first threshold exceeds athird proportion higher than the first proportion, causing a sixthprocessor which is stopped among the plurality of processors, to beoperated; and causing the sixth processor to be coupled with a thirdstorage device which is not coupled with the plurality of processors andthe virtual machines among the plurality of storage devices.
 15. Thenon-transitory computer-readable storage medium according to claim 12,wherein the process further comprising: when a frequency of a processingload of a seventh processor among the plurality of processors being lessthan a third threshold lower than the first threshold exceeds a fourthproportion, causing an eighth processor which does not operate thevirtual machine among the plurality of processors, to be stopped;causing a third virtual machine to be operated on a ninth processorwhich is operating among the plurality of processors; and causing thefirst storage device to be coupled with the operated third virtualmachine.
 16. The non-transitory computer-readable storage mediumaccording to claim 15, wherein the process further comprising: when afrequency of a processing load of a tenth processor among the pluralityof processors being less than the third threshold exceeds a fifthproportion higher than the fourth proportion, causing data which is heldin a fourth storage device coupled with a fourth virtual machine whichis operating on an eleventh processor among the plurality of processorsto be transmitted to a fifth storage device which is coupled with anoperating fifth virtual machine among the plurality of virtual machineor coupled with an operating twelfth processor among the plurality ofprocessors; causing the fourth virtual machine to be terminated; andseparating the fourth storage device from the fourth virtual machine.